High dynamic range video

ABSTRACT

Certain cameras and systems described herein produce enhanced dynamic range still or video images. The images can also have controlled or reduced motion artifacts. Moreover, the cameras and systems in some cases allow the dynamic range and/or motion artifacts to be tuned to achieve a desired cinematic effect.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims benefit of priorityunder 35 U.S.C. §120 to U.S. patent application Ser. No. 13/215,002,filed Aug. 22, 2011, entitled “HIGH-DYNAMIC RANGE VIDEO”, and claimsbenefit of priority from U.S. Provisional Application Nos. 61/376,172,filed Aug. 23, 2010, and 61/473,711, filed Apr. 8, 2011. The disclosuresof each of the foregoing applications are incorporated in their entiretyby reference herein.

BACKGROUND

The disclosure herein relates to devices for creating improved dynamicrange images, such as from images of still or moving scenes detectedwith a digital camera.

The dynamic range of a digital image can be characterized as the ratioin intensity between the brightest and the darkest measured values inthe image. Thus, an image having a wide dynamic range represents moreaccurately the wide range of intensity levels found in real scenes.

SUMMARY

Conventional digital cameras and techniques typically produce relativelylow dynamic range images and provide limited flexibility in controllingdynamic range. Dynamic range can be limited by factors including thelevel of brightness detectable by each sensor element (e.g., thebit-width of each pixel) and the level of exposure. For example, acaptured image may accurately represent the intensity of relatively darkregions of the scene, but not relatively bright regions of the samescene, or vice versa.

Different techniques can be used to enhance the dynamic range of images.However, particularly for moving images, existing techniques aregenerally overly complex, provide sub-optimal image quality, or allowlimited creative flexibility. For example, some techniques producerelatively large, un-controlled motion artifacts (e.g., blurring orflickering). For these and other reasons, certain cameras and systemsdescribed herein produce enhanced dynamic range images also havingcontrolled or reduced motion artifacts. Moreover, according to someaspects, the system provides an interface allowing users to tune thedynamic range and/or motion artifacts to achieve a desired cinematiceffect.

According to certain aspects, a method is provided of obtaining imagedata using a digital imaging sensor comprising a picture element arrayof rows and columns. The method can include using a plurality of pictureelements in the picture element array to capture light corresponding toa first image at a first exposure level and using the plurality ofpicture elements in the picture element array to capture lightcorresponding to a second image at a second exposure level differentthan the first exposure level. The method can further include convertingthe light captured by the plurality of picture elements for the firstimage to digital measurements and converting the light captured by theplurality of picture elements for the second image to digitalmeasurements. In some embodiments, the step of using the plurality ofpicture elements in the picture element array to capture lightcorresponding to the second image at the second exposure level beginsprior to the completion of the step of converting the light captured bythe plurality of picture elements for the first image to digitalmeasurements. Moreover, in certain embodiments, the step of convertingthe light captured by the plurality of picture elements for the firstimage to digital measurements substantially completes prior to thebeginning of the step of converting light captured by the plurality ofpicture elements for the second image to digital measurements.

According to additional aspects, an imaging system is provided,comprising a plurality of picture elements arranged in an array. Theimaging system can further comprise control circuitry configured tocapture light corresponding to a first image at a first exposure levelwith the plurality of picture elements in the picture element array andcapture light corresponding to a second image at a second exposure leveldifferent than the first exposure level with the plurality of pictureelements in the picture element array. The control circuitry can furtherbe configured to convert the light captured by the plurality of pictureelements for the first image to digital measurements and convert lightcaptured by the plurality of picture elements for the second image todigital measurements. According to some embodiments, the controlcircuitry begins the capturing of the light corresponding to the secondimage at the second exposure level prior to the completion of theconverting of the light captured for the first image to digitalmeasurements. The control circuitry may additionally substantiallycomplete the converting of the light for the first image to digitalmeasurements prior to the beginning of the converting of the light forthe second image to digital measurements.

According to further aspects, a method is provided of processing imagedata. The method can include receiving first image data corresponding toa plurality of picture elements in a picture element array used tocapture light corresponding to a first image at a first exposure level.The method may further include receiving second image data correspondingto the plurality of picture elements in the picture element array usedto capture light corresponding to a second image at a second exposurelevel. The method can additionally include selectively combining thefirst image data and the second image data to create combined image datacorresponding to a combined image having a wider dynamic range thaneither of the first image or the second image. In some cases, the usingthe plurality of picture elements in the picture element array tocapture light corresponding to the second image at the second exposurelevel began prior to the completion of converting the light captured bythe plurality of picture elements for the first image to digitalmeasurements. Moreover, the converting the light captured by theplurality of picture elements for the first image to digitalmeasurements substantially completed prior to the beginning of the stepof converting light captured by the plurality of picture elements forthe second image to digital measurements.

In yet further embodiments, a method is provided of obtaining image datausing a digital imaging sensor. The method can include capturing a firstimage with the sensor at a first exposure level and capturing a secondimage with the sensor at a second exposure level different from thefirst exposure level. The method may additionally include outputting thefirst image and outputting the second image. The capturing of the firstimage in some cases begins before said capturing of the second imagebegins. Additionally, the capturing of the first image and saidcapturing of the second image overlap in time in some cases. Further,the outputting of the first image and said outputting of the secondimage do not overlap in time in some embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example imaging systemaccording to embodiments described herein.

FIG. 2 is a block diagram illustrating an example sensor architecturecapable of generating multiple, closely-timed exposures according tosome embodiments.

FIGS. 3A-3D are timing diagrams showing operation of example sensorsrecording multiple, closely-timed exposures for multiple frames ofrecorded video in accordance with certain embodiments.

FIG. 3E is a timing diagram showing operation of a sensor operatingwithout recording closely-timed exposures.

FIGS. 4A-4C are timing diagrams showing operation of embodiments ofsensors recording closely-timed exposures for several rows of an imagesensor.

FIG. 5 is a block diagram illustrating an example process for blendingtracks of closely timed exposures having different exposure levels inaccordance with some embodiments.

FIGS. 6A-6B illustrate example operations for blending tracks ofclosely-timed exposures having different exposure levels in accordancewith certain embodiments.

FIG. 7 illustrates another example process for blending tracks havingdifferent exposure levels in accordance with some embodiments.

FIG. 8 illustrates an embodiment of a process for blending tracks ofclosely-timed exposures having different levels of exposure to controlmotion artifacts.

DETAILED DESCRIPTION

According to certain aspects, a camera system is provided having asensor capable of sequentially capturing and outputting first and secondimage(s) having different exposure levels. Moreover, the system beginscapturing the first and second (or more) images within a single frameperiod, providing controllable or reduced motion artifacts, among otheradvantages.

For instance, the first and second images are closely timed with respectto one another. Moreover, while there can be some separation in timebetween when the system captures pixels for the first image and capturescorresponding pixels for the second image, the first and second imagesmay nonetheless be referred to as or “conjoined” with one another intime because the separation does not result in visually significantartifacts when the images are combined. To achieve this effect, theexposure times for at least some portions of the first and second imagesoverlap with one another, in some configurations. When recording videoimages, some sensors disclosed herein output at least first and secondclosely-timed (or “conjoined”) image streams (also referred to herein as“tracks”). According to additional aspects, systems and methodsdescribed herein process the first and second image streams (e.g.,in-camera or during post-processing) to create a combined output stream.

For example, for each video frame, the imaging system capturesindividual exposures in a second (e.g., relatively high exposure) trackvery soon after the capturing corresponding exposures in the first(e.g., relatively low exposure) track. In this manner, certaintechniques described herein reduce or remove the potential for visualseparation between objects in an image track having a first exposurelevel as compared to the same objects in an image track(s) having adifferent exposure level. Undesirable levels of visual separationbetween objects could otherwise arise between the objects in the firstimage and corresponding objects in the second image, due either tomovement of objects in scene or of the camera, for example.

Moreover, the system acquires the individual images from the sensorrelatively quickly. Where the system incorporates a rolling shutter, forexample, the period of delay from between beginning exposure for thefirst rows of the sensor for a particular image and beginning theexposure for the last rows of the sensor for the image is relativelyshort. Similarly, the period of delay from between reading out the firstrows for an image to reading out the last rows of the sensor for theimage is also relatively short. In this manner, the system limits theamount of motion artifact (e.g., wobble, skew, smear) introduced by therolling shutter. This is in contrast to configurations where the amountof time it takes to capture and read out an entire exposure consumes alarger period of time.

Moreover, systems described herein according to some aspects exploit theminimal gap between first and second images, limited rolling shutterartifacts, and other aspects of the techniques described herein toachieve a variety of desirable creative effects. For example, the systemcan selectively blend or combine the closely-timed image streams toadjust the dynamic range, the character of the motion effect (e.g.,amount or quality of blur), or other characteristic.

In some cases, the blending of the image streams is tailored to achievea motion effect substantially similar to how motion is perceived by thehuman eye. In some instances, the blending is adjusted such that theblurring effect is similar to that of a traditional camera. A user mayadvantageously control the amount or quality of the image streamblending, providing additionally creative power and flexibility. Suchprocesses can be partially or fully automated, as well.

Thus, the techniques described herein provide a variety of benefits.Particularly in the area of recording motion video, certain embodimentsprovide recordings having improved-dynamic range, customized motion orother visual effects, and combinations thereof.

While the techniques described herein are discussed primarily withrespect to digital motion cameras recording continuous video, it will beunderstood that various aspects of the inventions described herein arecompatible with still cameras, as well as digital still and motioncameras (DSMC's).

System Overview

FIG. 1 is a block diagram an example system 100 that detects opticaldata and processes the detected data. The system 100 can include animaging device 101 that can include an optics module 110, and an imagesensor 112, which may further include a multiple exposure generator 113.The device 101 can also include a memory 114 and an image processingmodule 116. The image processing module 116 may in turn include ablending module 117 that is generally configured to adjust the dynamicrange of an output image or stream of images. The blending module 117may further be configured to adjust the motion effect in the outputstream.

The imaging device 101 may be a camera system, for example, and theoptics module 110, image sensor 112, memory 114, and the imageprocessing module 116 or portions thereof may be housed in or supportedby the camera housing. Moreover, portions of the system 100 may beincluded in a separate device. For example, as shown, the blendingmodule 117 or portions thereof in some configurations resides in aseparate computing device 102, such as a laptop or other computer.

The optics module 110 focuses an image on the image sensor 112. Sensors112 may include, for example, an array of charge-coupled devices (CCD)or Complementary Metal-Oxide-Semiconductor (CMOS) image sensor cellssuch as active-pixel sensor cells. Such image sensors are typicallybuilt on silicon chips and may contain millions of image sensor cells.Each sensor cell detects light reaching its surface and outputs a signalthat corresponds to the intensity of the light detected. The detectedlight is then digitized.

Because these image sensors are sensitive to a broad spectrum ofwavelengths of light, a color filter array can be disposed over thelight sensitive surface of such sensors. One type of color filter arrayis a Bayer pattern color filter array, which selectively passes red,blue, or green wavelengths to sensor elements. The output of such asensor, however, is a mosaic image. This mosaic image is formed by theoverlapping matrices of red, green, and blue pixels. The mosaic image isusually then demosaiced, so that each picture element has a full set ofcolor image data. The color image data may be expressed in the RGB colorformat or any other color format.

Some of the embodiments disclosed herein are described in the context ofa video camera having a single sensor device with a Bayer patternfilter. However, the embodiments and inventions herein can also beapplied to cameras having other types of image sensors (e.g., CMY Bayeras well as other non-Bayer patterns), other numbers of image sensors,operating on different image format types, and being configured forstill and/or moving pictures. It is to be understood that theembodiments disclosed herein are exemplary but non-limiting embodiments,and the inventions disclosed herein are not limited to the disclosedexemplary embodiments.

The optics hardware 110 can be in the form of a lens system having atleast one lens configured to focus an incoming image onto the imagesensor 112. The optics hardware 110, optionally, can be in the form of amulti-lens system providing variable zoom, aperture, and focus.Additionally, the optics hardware 110 can be in the form of a lenssocket supported by a camera housing and configured to receive aplurality of different types of lens systems for example, but withoutlimitation, the optics hardware 110 include a socket configured toreceive various sizes of lens systems including a 50-100 millimeter (T3)zoom lens, a 50-150 millimeter (T3) zoom lens, an 18-50 millimeter (T3)zoom lens, an 18-85 millimeter (T2.9) zoom lens, a 300 millimeter (T2.8)lens, 18 millimeter (T2.9) lens, 25 millimeter (T1.8) lens, 35millimeter (T1.8) lens, 50 millimeter (T1.8) lens, 85 millimeter (T1.8)lens, 85 millimeter (T1.8) lens, 100 millimeter (T1.8) and/or any otherlens or any other lens. In certain embodiments, a 50-100 millimeter(F2.8) zoom lens, an 18-50 millimeter (F2.8) zoom lens, a 300 millimeter(F2.8) lens, 15 millimeter (F2.8) lens, 25 millimeter (F1.9) lens, 35millimeter (F1.9) lens, 50 millimeter (F1.9) lens, 85 millimeter (F1.9)lens, and/or any other lens. As noted above, the optics hardware 110 canbe configured such that despite which lens is attached thereto, imagescan be focused upon a light-sensitive surface of the image sensor 112.

The image sensor 112 can be any type of image sensing device, including,for example, but without limitation, CCD, CMOS, vertically-stacked CMOSdevices such as the Foveon® sensor, or a multi-sensor array using aprism to divide light between the sensors. In some embodiments, theimage sensor 112 can include a CMOS device having about 12 millionphotocells. However, other size sensors can also be used. In someconfigurations, camera 10 can be configured to output video at “5 k”(e.g., 5120×2700), Quad HD (e.g., 3840×2160 pixels), “4.5 k” resolution(e.g., 4,520×2540), “4 k” (e.g., 4,096×2,540 pixels), “2 k” (e.g.,2048×1152 pixels) or other resolutions. As used herein, in the termsexpressed in the format of xk (such as 2 k and 4 k noted above), the “x”quantity refers to the approximate horizontal resolution. As such, “4 k”resolution corresponds to about 4000 or more horizontal pixels and “2 k”corresponds to about 2000 or more pixels.

The camera can also be configured to downsample and subsequently processthe output of the sensor 112 to yield video output at 2K, 1080 p, 720 p,or any other resolution. For example, the image data from the sensor 112can be “windowed”, thereby reducing the size of the output image andallowing for higher readout speeds. However, other size sensors can alsobe used. Additionally, the camera can be configured to upsample theoutput of the sensor 112 to yield video output at higher resolutions.Moreover, the camera can be configured to capture and record video at10, 20, 24, 30, 60, and 120 frames per second, or any other frame rate.Additionally, the blending techniques described herein are generallycompatible with a variety resolutions and frame rates, including thoselisted above.

The memory 114 can be in the form of any type of digital storage, suchas, for example, but without limitation, hard drives, solid-statedrives, flash memory, optical discs, or any other type of memory device.In some embodiments, the size of the memory 114 can be sufficientlylarge to store image data from the compression module corresponding toat least about 30 minutes of video at 12 mega pixel resolution, 12-bitcolor resolution, and at 60 frames per second. However, the memory 114can have any size.

In some embodiments, the memory 114 can be mounted on an exterior of acamera housing. Further, in some embodiments, the memory can beconnected to the other components through standard or customcommunication ports, including, for example, but without limitation,Ethernet, USB, USB2, USB3, IEEE 1394 (including but not limited toFireWire 400, FireWire 800, FireWire S3200, FireWire S800T, i.LINK, DV),SATA and SCSI. Further, in some embodiments, the memory 114 can comprisea plurality of hard drives, such as those operating under a RAIDprotocol. However, any type of storage device can be used.

The image processing module 116 may, for example, operate on data thatis stored in the memory 114. Alternatively, the image processing modulemay operate on data as it comes from the image sensor 112, and theprocessed data may then be stored in the memory 114. The imageprocessing module 116 may perform a variety of operations on the data.Depending on the operation, the processing module 116 may perform theoperation on the individual first and second image streams, prior toblending, or alternatively, on the blended HDR output stream.

For example, the image processing module 116 may compress the data fromthe image sensor, perform pre-compression data-preparation (e.g.,pre-emphasis and/or entropy reduction), format the compressed data, andthe like. Examples of such techniques are described in greater detail inU.S. Pat. No. 7,830,967 entitled “VIDEO CAMERA” (the '967 patent), whichis incorporated by reference herein in its entirety. In some instances,the image processing module 116 processes the individual image stream(or combined image stream) to tune the image streams to perceptualspace, as will be described in detail below with respect to FIG. 8A.Additional compatible techniques include green data modification andpre-emphasis processes shown and described throughout the '967 patent(e.g., with respect to FIGS. 8-11 and columns 11-13). In general,certain embodiments described herein are compatible with and/or arecomponents of embodiments described in the '967 patent. Thus, some orall of the features described herein can be used or otherwise combinedwith various features described in the '967 patent, including featuresdescribed in the '967 patent which are not explicitly discussed herein.

The sensor 112 can be configured to sequentially capture and output atleast first and second image streams (or tracks) having differentexposure levels. For example, the multiple exposure generator 113 maycomprise specially designed timing control logic that controls thesensor 112 to output the image streams. In some cases, the sensor 112can output more than two tracks (e.g., totaling 3, 4, 5, 10 or more)having various exposure levels.

Where there is a significant amount of delay between exposures orportions of exposures in a first track and corresponding exposures orportions of exposures in a second track, undesirable motion artifactscan arise. For example, such artifacts can arise where an object in therecorded scene (e.g., an actor or other photographic subject) moves avisually significant distance between the time the first image streamframe is captured and the time the second image stream frame iscaptured. In such cases, visual separation, or “gaps,” can arise betweenobjects in one track as compared to the same object in another track.Such issues can be difficult to correct for or control inpost-processing.

For these and other reasons, the sensor 112 according to certain aspectsis configured to sequentially output at least first and second imagestreams having different exposure levels and which are closely timedwith respect to one another. For example, the multiple exposuregenerator 113 may control a pixel array and output circuitry of thesensor 112 to cause the sensor 112 to output first and second imagestreams. Each image stream can have multiple frames which areclosely-timed and/or at least partially overlapping in time withcorresponding frames from the other image stream. For instance, theamount of time from between when the system converts captured data forpicture elements in the images from the first stream to digital valuesand begins to expose the same picture elements for corresponding imagesin the second image stream is relatively short. Moreover, in some cases,the multiple exposure generator 113 captures at least some of thepicture elements for the images in the second stream while capturingother picture elements for the corresponding images in the first stream,during overlapping time periods. In this manner, the visual separationbetween moving objects in the first image stream and the same movingobjects in the second image stream may be relatively insignificant, ormay be otherwise substantially reduced. The multiple exposure generator113 may generally be a specialized timing generator coupled to thesensor 112, and is described in greater detail with respect to FIG. 2.

Because there may be no significant visual separation between objects ina frame in the first image track frame and the corresponding objects ina subsequently captured frame in the second image track, the first andsecond image tracks may also be referred to as having no “gaps” withrespect to one another, or as substantially “touching” one another. Forexample, while there may be some minor delay between the capture ofpicture elements for first images and the capture of picture elements inthe second images, the delay may be so small that there is reduced orsubstantially no visual object separation at full sensor resolution, atone or more scaled-down resolutions, or a combination thereof. Due tothe closely-timed nature of the tracks, the motion artifacts in thecombined output track are substantially reduced, removed, or can becontrolled in a desired fashion.

Sensors described herein are referred to herein as having “adjacentread” or “conjoined exposure” capability. Additionally, the imagestracks may be referred to as being “adjacent,” “closely timed,”“conjoined,” or the like.

Moreover, as described above, the individual exposures according to someembodiments are captured and/or output in a compact manner, makingefficient use of the frame period.

The blending module 117 according to certain embodiments selectivelycombines the closely-timed tracks to produce an output track having adesired visual effect. For instance, the output video stream can have anenhanced dynamic range relative to the dynamic range of the individualfirst and second image tracks. As an example, the first image stream mayhave a relatively low exposure level (e.g., short integration time),providing useful data in relatively bright regions of a captured scene,such as highlight detail. The second image stream may have a relativelyhigher exposure level (e.g., longer integration time), providing usefuldata in relatively darker regions of the scene, such as detail in theshadows.

Moreover, in some cases where the sensor 112 outputs more than twotracks, the blending module 117 is capable of combining a correspondingnumber of tracks. Where the blending module 117 or portions thereofreside in the external computing device 102, the dynamic range and/ormotion processing can be done in post-processing. Additionally,depending on the implementation, the blending module 117 may operate ondata stored in the memory 114, on data as it is output from the imagesensor 112, or on data stored in the external computing device 102.

As discussed, the combined output stream may be created on camera oralternatively off camera, during post-processing. In one configuration,the user can select on-camera or off camera creation of the outputstream, as desired. Moreover, as will be described in further detailbelow with respect to FIGS. 5-9, in various embodiments the blendingmodule 117 can combine the closely-timed tracks according to a varietyof algorithms. The algorithms can be fixed or user-selectable oradjustable. For example, in one embodiment, the image streams arecombined on-camera according to a fixed algorithm. In anotherconfiguration, the user selects from a variety of algorithms, dependingon the desired creative effect.

Generating Tracks of Closely-Timed Images Having Multiple ExposureLevels

FIG. 2 is a block diagram illustrating example image sensor 200 capableof generating closely-timed exposures according to embodiments describedherein. Referring to FIG. 2, the image sensor 200 includes a pixel array202, output circuitry 204, and a multiple exposure generator 206. Aswill be described in greater detail, the multiple exposure generator 206generally controls the various components of the sensor 200 to provideclosely-timed exposures on the output bus 212 of the sensor 200 forfurther storage and processing.

The image sensor 200 may be similar to the image sensor 12 describedabove with respect to FIG. 1, for example, and the pixel array 202includes a plurality of pixels (e.g., photocells) arranged in a matrixincluding “M” rows and “N” columns. While a wide variety of values arepossible for “M” and “N”, an example “5 k” sensor includes 5,120 rowsand 2,700 columns, an example Quad HD sensor includes 3,840 and 2,160pixels, an example “4.5 k” resolution sensor includes 4,520 rows and2,540 columns, an example“4 k” sensor includes 4,096 rows and 2,540columns, and an example “2 k” sensor includes 2,048 and 1,152 columns.

In the illustrated example, sensor 200 is configured to output a singlerow at a given time, and the sensor 200 includes one instance of outputcircuitry 204 that is configured to process and output image informationfor a single row. In another embodiment, such as where a Bayer patternsensor is used, the sensor outputs two rows at a time and can includetwo instances of the output circuitry 204. In yet other configurations,the sensor 200 outputs other numbers of rows during a given time slot(e.g., 2, 3, 4, 5, 10 or more rows), and can include a correspondingnumber of instances of the output circuitry 204.

During operation, the multiple exposure generator 206 provides rowselect information via input bus 208 to row decode logic (not shown) ofthe pixel array 202. For each of the selected rows, the pixel array 202provides the stored values for the N pixels corresponding to the columnsof the selected row (or subset of rows) to the output circuitry 204.

The output circuitry 204 of the example sensor 200 also receives timingcontrol signals on the bus 210 from the multiple exposure generator 206,and is generally configured to process and digitize the analog pixelvalues received from the pixel array. The output circuitry 204 of theexample sensor 200 includes sets of programmable-gain amplifiers (PGAs)and analog-to-digital converters (ADCs), although a variety componentsmay be used in various implementations. The output circuitry 204 in turnpresents the digitized, processed values of the currently selected rowon the output bus 212. For example, the sensor 200 may transmit thevalues to the memory 114, image processing module 116, or othercomponents of the system of FIG. 1 for storage and processing. In someinstances, the sensor buffers the values for one or more rows beforetransmission on the output bus 212.

As shown in FIG. 2, the multiple exposure generator 206 may resideoutside of the packaging 214. For example, the packaging 214 is anintegrated circuit packaging and the pixel array 202 and outputcircuitry 204 are formed on an integrated circuit housed in thepackaging 214. The multiple exposure generator 206, on the other hand,may form a part of a separate integrated circuit that is housed in aseparate package. In another embodiment, the multiple exposure generator206 is included on the same integrated circuit as the other components,and is housed in the sensor package 214.

In some cases, the multiple exposure generator 206 comprises logicimplemented in a field-programmable gate array (FPGA), although themultiple exposure generator 206 can be implemented in a variety of ways,and can include hardware, software, analog and/or digital circuitry,such as custom circuitry, a microprocessor, an application-specificintegrated circuit (ASIC), combinations thereof and the like.

FIG. 3A is an example timing diagram 300 illustrating two consecutiveframes of video for a camera that is recording two closely-timed imagetracks (also referred to as image streams) in accordance with anembodiment. Two closely-timed exposures 302, 304 are taken within eachvideo frame, one for each image track. Thus, a first image trackcorresponds to the first exposures 302 taken over a series of recordedvideo frames and a second image track 304 corresponds to secondexposures 304 taken over the same series of video frames.

The camera exposes the first exposure 302 for each frame for a firstintegration period 306, and exposes the second exposure 304 for eachframe for a second integration period 308. The timing diagram 300 may berepresentative of the operation of the sensors 112, 200 of FIGS. 1 and2, for example. Moreover, while the diagram shows operation for only twoframes for the purposes of illustration, it will be appreciated that thecamera may record any number of frames, depending on the length of therecording.

As shown, the camera of the illustrated example exposes the rows 0→M ofthe sensor in an overlapping fashion. In this manner, the first row (orsubset of rows) is exposed for a first integration period beginning attime t₀ and ending at time t₁. The remaining rows (or subsets of rows)are exposed in successive, overlapping time slots, and the firstintegration period for the last row extends from time t₂ to time t₃.This general exposure scheme, where successive rows (or subsets of rows)are exposed in successive, overlapping time slots, may generally bereferred to as a “rolling shutter” technique.

Picture elements for the second exposure 304 are advantageously capturedrelatively shortly after capturing the same picture elements for thefirst exposure 302. For example, the inter-exposure delay 312 betweenthe end of the first exposure period 306 for each row (or subset ofrows) and the beginning of the second exposure period 308 for the samerow is relatively minimal. For example, as discussed in greater detailwith respect to FIG. 4A below, the inter-exposure delay 312 may dependin one instance on the reset time for one or more rows of the sensor. Inother cases, the inter-exposure delay 312 can depend on some otherparameter(s). Referring again to the sensor 200 of FIG. 2, the multipleexposure generator 206 is configured to generate control signals foroperating the pixel array 202 and output circuitry 204 to generate theclosely-timed exposures 302, 304.

Moreover, the inter-row skew for the individual exposures 302, 304introduced by the rolling shutter is relatively minimal. This results ina shorter overall image acquisition time, and helps to control anymotion artifacts that may otherwise be introduced by the rollingshutter. As will be discussed further with respect to FIG. 4A, theinter-row skew and image acquisition time can be related to the readouttime for a row (or subset rows).

As shown, the system additionally makes efficient use of the sensorhardware and frame period by overlapping certain portions of the firstand second exposures in time. For instance, the system begins to exposeat least some of the rows (or subsets of rows) for the second exposure304 before all of the rows have been converted and/or read out for thefirst exposure 302. Moreover, at least some rows are exposed for thefirst exposure 302 concurrently with other rows for the second exposure304.

The system in certain embodiments additionally substantially completesthe conversion and/or readout of the sensor rows (or subsets of rows)for the first exposure 302 before beginning to convert and/or readoutthe rows for the second exposure 304. By dedicating the conversion andreadout circuitry to each of the respective exposures until they arecomplete, the system helps maintain a compact overall acquisition timefor the individual exposures 302, 304. In the illustrated embodiment,the system completes converting and/or reading out 100 percent of thesensor rows for first exposure 302 before beginning to convert and/orreadout any of the rows for the second exposure 304. In other cases, thesystem completes converting and/or reading out at least 80 percent, atleast 90 percent, at least 95 percent, at least 96 percent, at least 97percent, at least 98 percent or at least 99 percent of the sensor rowsfor the first exposure 302 before beginning to convert and/or read outthe rows for the second exposure 304. In yet further cases, the systemsubstantially completes conversion and/or readout of the rows for thefirst exposure 302 before reading out any of the rows for the secondexposure 304, or before reading out more than 1 percent, more than 2percent, more than 3 percent, more than 4 percent, more than 5 percent,more than 10 percent, or more than 20 percent of the rows for the secondexposure 304, depending on the embodiment. Further details regarding thespecifics of the sensor timing are provided below with respect to FIGS.4A-4C.

Depending on the particular configuration, the duration of the varioustime periods shown in FIG. 3A can vary greatly, including the frameperiod 312, first and second exposure integration times 302, 304,inter-exposure delay 312 and sensor inactive 310. The sensor inactivetime 310 may refer to a period where sensor elements are substantiallyinactive and not being exposed. For example, when capturing the firstand second exposures, the sensor electronically activates theappropriate picture elements according to the rolling shutter techniqueto be sensitive to and capture incident light. In contrast, theappropriate pixel elements are electronically deactivated and thereforenot sensitive to incident light during the sensor inactive period 310.In other implementations, where a physical shutter is used, the physicalshutter is opened/closed in place of the electronicactivation/deactivation. The sensor inactive period can also instead bedefined as the period of time from between t₇, when exposure completesfor the last row in the second exposure 304, to the end of the frameperiod, immediately before the first row in the first exposure 302begins to expose for the next frame.

FIGS. 3B-3D illustrate timing diagrams for several exampleconfigurations. For clarity, the inter-exposure delay 312 is not shownin FIGS. 3B-3D. In one embodiment, FIG. 3B illustrates a scenario wherethe frame period is about 41.7 milliseconds (24 fps), the firstintegration time 306 is about 2.6 milliseconds, the second integrationtime 308 is about 20.8 milliseconds, the inter-exposure delay 312 isabout 15.6 microseconds, and the shutter closed time is about 18.2milliseconds. The second integration time 308 is about half the frameperiod, and thus generally corresponds to an integration time for asensor implementing a 180 degree shutter. In another example case, FIG.3C shows a scenario where the frame period 312 is about 41.7milliseconds (24 fps), the first integration time 306 is about 2.6milliseconds, the second integration time 308 is about 31.6milliseconds, the inter-exposure delay 312 is about 15.6 microseconds,and the shutter closed time 310 is about 7.4 milliseconds. In this case,the second integration 308 time corresponds to an integration time for asensor implementing a 273 degree shutter. In yet another instance, FIG.3D illustrates an case where the frame period 312 is about 41.7milliseconds (24 fps), the first integration time 306 is about 0.65milliseconds, the second integration time 308 is about 5.2 milliseconds,the inter-exposure delay 312 is about 15.6 microseconds, and the shutterclosed time 310 is about 35.75 milliseconds. Here, the secondintegration time 308 corresponds to an integration time for a 45 degreeshutter.

For comparison, FIG. 3E illustrates a timing diagram for another examplecamera capturing multiple exposures having different exposure levels.However, unlike the embodiments shown in FIGS. 3A-3D, the camera of FIG.3E captures the long exposures 304 and short exposures 306 on different,alternating frames. Thus, the exposures 304 and 306 are notclosely-timed, and significant undesirable motion artifacts may bepresent. For example, relatively large gaps of visual separation mayexist between image scene objects in the long exposures 304 as comparedto image scene objects in the short exposure 306 in the next frame.Thus, it can be difficult in such cases to combine the short and longexposures 304, 306 to create high-dynamic range images with controlledmotion artifacts.

FIG. 4A shows a timing diagram 400 illustrating closely-timed first andsecond exposures 402, 404 for one frame of recorded video. For clarity,the timing diagram is shown for only the first three rows R1, R2, R3 (orsubsets of rows) of an image sensor. For example, the diagram 400 maycorrespond to one embodiment of a more detailed depiction of theexposure timing for the first three rows of the sensor whose operationis depicted in any of FIGS. 3A-3D. Referring to the first row R1, beforethe first exposure 402 begins, the row R1 is initially reset, asindicated by the vertical arrow labeled “RST.”

After a reset period 410 passes corresponding to the amount of time ittakes to reset the pixels in the row R1, the photocells in the row R1begin to accumulate charge during the first integration period 406.Following the first integration period 406, the pixels in the first rowR1 have accumulated charge corresponding to the first exposure level. Areadout cycle is then initiated for the row R1, along with another resetcycle, as indicated by the vertical arrow labeled “RO/RST”. Referring toFIG. 2, at the end of the readout period 412, the row R1 is readout ofthe pixel array 202 by the output circuitry 204, and is provided on theoutput 212 for storage and/or processing.

Additionally, once the row R1 has completed the second reset cycle, thesecond exposure 404 begins. As such, the inter-exposure delay 410between the first and second exposures in one embodiment depends on thereset time 410 for a row of pixels (or subset of rows where more thanone row are read out at once). For example, the inter-exposure delay 410may correspond to, substantially correspond, or be approximately equalto the reset time 410 for one row (or subset of rows). In otherinstances, the inter-exposure delay 410 is less than or equal toapproximately one, approximately 2, approximately 5, approximately 10,approximately 20, approximately 50 or approximately 100 reset times 410for a row (or subset of rows). In yet further cases, the inter-exposuredelay 410 is some value between approximately one reset time 410 for arow (or subset of rows) and approximately 2, approximately 5,approximately 10, approximately 20, approximately 50 or approximately100 reset times, depending on the embodiment. As is described in greaterdetail below, the reset time 410 can advantageously be relatively short,leading to closely timed first and second exposures 402, 404.

In other configurations, the delay between the first and secondexposures 402, 404 can correspond to a variety of values other than thereset time 410. For example, the inter-exposure delay 410 may correspondto, substantially correspond to, or otherwise depend on the readout time412, such as where the readout time is greater than or equal to thereset time. In one such embodiment, the inter-exposure delay 410 isapproximately equal to the readout time 412. In other cases, dependingon the specific implementation, the inter-exposure delay 410 is lessthan or equal to approximately one, approximately 2, approximately 5,approximately 10, approximately 20, approximately 50 or approximately100 readout times 412 for a row (or subset of rows). According to yetfurther embodiments, the inter-exposure delay 410 is be some valuebetween approximately one readout time 412 for a row (or subset of rows)and approximately 2, approximately 5, approximately 10, approximately20, approximately 50 or approximately 100 readout times 412. In furthercases, the delay 410 between the first and second exposures 402, 404depends on both the readout time and the reset time. For example, thereset operation in one embodiment is initiated only after completion ofthe readout operation. In this case, the delay between the first andsecond exposures 402, 404 may correspond to or substantially correspondto the sum of the readout time 412 and the reset time. In such cases,the inter-exposure delay 410 may be less than or equal to approximatelythe sum of the reset time 410 and the readout time 412 for a row (orsubset of rows), or may alternatively be less than or equal toapproximately the sum of the reset time 410 and the readout time 412,multiplied by a factor of 2, 5, 10, 20, 50 or 100. In furtherembodiments, the inter-exposure delay 410 can be some value betweenapproximately the sum of the reset time 410 and the readout time 412 fora row (or subset of rows) and approximately the sum of the reset time410 and the readout time 412, multiplied by a factor of 2, 5, 10, 20, 50or 100.

As shown, the sensor initiates a similar sequence of operations for theother rows R2, R3 . . . RM. However, as described above, with respect toFIG. 2, the sensor may be configured to output only one row (or a subsetof two or more rows) at a time. Thus, for each subsequent row, themultiple exposure generator 206 waits at least one readout period 412before initiating the initial reset cycle. As such, the exposure timingfor each row is staggered in time with respect to the other rows. In theillustrated example, each subsequent row is delayed by the readoutperiod 412 with respect to the preceding row. As such, when anintegration period 402, 404 is complete for a particular row, the outputcircuitry 204 will be generally immediately available. In this manner,the utilization of the readout circuitry (e.g., the output circuitry 204of FIG. 2) is relatively high. Referring again to FIG. 2, the multipleexposure generator 206 may send the appropriate control signals to thepixel array via the bus 208 and to the output circuitry 204 via the bus210, initiating the reset and readout of the rows R1, R2, R3 . . . RM inthe manner described.

Although a variety of values are possible, one example sensor 200 has2740 rows of pixels configured to be readout two rows at a time, and thereadout time 412 for the two rows of the example sensor 200 is about 7.4microseconds. In this example, the reset time 410 for resetting a subsetof two rows is about 15.6 microseconds. Thus, the time between the firstand second exposures 402, 404 and the time between the closely-timedimage streams is about 15.6 microseconds, or about 0.03 percent of theframe period for a frame rate of 24 fps, and about 0.05, 0.08, 0.09, and0.16 percent of the frame period for frame rates of 30, 50, 59.94, and100 fps, respectively. In another configuration, for example, the sensor200 produces first and second image streams which are temporally spacedby no more than about 7 microseconds. As such, for each frame the sensor200 captures the a subsequent 404 (corresponding to the second imagestream) no more than about 7 microseconds after capturing the precedingexposure 402 for that frame (corresponding to the first image stream).In other embodiments, the first and second exposures 402, 404 andcorresponding image streams are temporally spaced by no more than about4 microseconds, no more than about 2 microseconds, or no more than about1 microsecond. In yet other embodiments, the first and second exposures402, 404 are spaced by no more than about 1 millisecond, no more thanabout 100 microseconds, no more than about 50 microseconds, no more thanabout 25 microseconds, no more than about 20 microseconds, no more thanabout 16 microseconds, no more than 15.6 microseconds, no more thanabout 15 microseconds, or no more than about 10 microseconds. Asindicated above, in some embodiments there are more than two exposureswithin each frame. In such cases, the delay between any two adjacentexposures may correspond to any of the above listed values or ranges ofvalues. In various embodiments, the inter-exposure delay is less thanabout 0.01, 0.02, 0.03, 0.04, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 20, or30 percent of the frame period.

Additionally, as mentioned previously, because acquisition of each row(or subset of rows) is staggered in time from the immediately precedingrow by a relatively minimal amount, the skew and resulting motionartifacts introduced by the rolling shutter are reduced. For example,the inter-row acquisition skew between the time the system begins toacquire image data for a row (or subset of rows) and begins to acquireimage data for the immediately subsequent row corresponds to the readouttime 412 for a row (or subset of rows) in the illustrated embodiment.Likewise, an inter-row readout skew between the time the systemcompletes reading out acquired image data for a row (or subset of rows)and completes reading out acquired image data for the subsequent row (orsubset of rows) corresponds to the readout time 412 for a row (or subsetof rows). In various embodiments, the inter-row acquisition skew and/orthe inter-row readout skew is approximately equal to one readout time412 for a row or subset of rows, or is less than or equal toapproximately one, approximately 1.5, approximately 2, approximately2.5, or approximately 3 readout times 412.

Extending the inter-row skew across all of the rows for FIG. 4A, thetotal intra-exposure acquisition skew across the entire image frombetween the time the system begins to acquire a first of the rows (orsubset of rows) and begins to acquire all of the rows or substantiallyall (e.g., at least 90, 95 or 99 percent) of the rows is given byapproximately:(M−1)*read_out_time,  (Eq. 1)

Similarly, the intra-exposure readout skew between the time the systemcompletes reading out a first of the rows and completes reading outsubstantially all of the rows may correspond to the same amount. As aspecific example, for the 2740 row sensor described above, theintra-exposure acquisition and readout skews are both equal to about10.1 milliseconds ((2740/2−1)*7.4 microseconds).

Further, the acquisition time for an entire exposure, from between thetime when the system begins to acquire a first of the rows (or subsetsof rows) and completes reading out substantially all of the rows for theexposure is approximately given by:exposure_time+[(M−1)*read_out_time].  (Eq. 2)

In further compatible embodiments, the image acquisition time is lessthan or equal to one of the following:exposure_time+[(M−1)*read_out_time]  (Eq. 3)exposure_time+[(M−1)*1.5*read_out_time],  (Eq. 4)exposure_time+[(M−1)*2*read_out_time],  (Eq. 5)exposure_time+[(M−1)*2.5*read_out_time], or  (Eq. 6)exposure_time+[(M−1)*3*read_out_time].  (Eq. 7)

The systems in each of the above-described specific examples record afirst relatively lower exposure (e.g., short exposure) followed by asecond relatively higher exposure (e.g., long exposure) for each frame.However, the number and order of the inter-frame exposures can vary, andcan be customized as desired. For example, in one instance, each frameincludes a long track followed by a short track. Additionally, thedevice can be configured to capture more than two exposures per frame,such as 3, 4, 5, 10 or more exposures. As just a few illustrativeexamples, the following capture patterns are possible including morethan two exposures per frame: (1) long, short, long; (2) short, long,short; (3) short, short+x, short+x*2, . . . , short+x*k; (4) long,long−x, long−x*2, . . . long−x*k; (4) short, medium, long; (5) long,medium, short; (6) medium, long, short.

The diagrams 300, 400 of FIGS. 3A-3D and 4A correspond to configurationsin which the sensor outputs the exposures in the same order for eachrow. For example, referring to FIG. 4, each of the rows R1 . . . RNoutputs the short exposure 402 followed by the long exposure 404 foreach frame. However, in some cases, the multiple exposure generator 206is configured to cause the sensor 200 to output the first and secondexposures in a different order depending on the row, providing greatercreative flexibility.

FIG. 4B shows one such example, where the order of the short and longexposures 402, 404 alternates from row to row (or from subset of rows tosubset of rows). Accordingly, the first row R1 outputs the shortexposure 402 followed by the long exposure 404, the second row R2outputs the long exposure 404 followed by the short exposure, the thirdrow outputs the short exposure 402 followed by the long exposure 404,and so on. In another similar embodiment, one or more of the integrationtimes for the short exposure and the long exposure for the odd rows aredifferent than the short exposure and the long exposure for the evenrows.

In yet other instances, one or more of the rows capture only a singleexposure per frame, while the remaining rows capture multiple exposures.For example, FIG. 4C shows one such case where the odd rows output asingle long exposure 402 per frame, while the even rows output threeclosely-timed exposures (e.g., short, medium, short) 404, 406, 408 perframe. Thus, it can be appreciated that a wide variety of possibilitiesexist for creating closely-timed exposures.

For the purposes of clarity, the terms “short” and “long” are usedherein to distinguish different exposures, such as where one exposurehas an integration time relatively shorter than another exposure(s), orshutter speed that is thus relatively longer than the other exposure(s).However, the above-described techniques are compatible with embodimentswhere other parameters instead of, or in addition to integration time,are manipulated and which affect the degree of exposure. Such parameterscan include gain (e.g., analog gain), aperture, and ISO, for example.Thus, depending on the embodiment, where exposures are referred to asbeing “short” or a “long,” it will be appreciated that they may actuallybe more accurately referred to as “low” or “high.” As used herein, theterms “picture element reset time” and “picture element readout time”can refer to one or more of the amount of time it takes to read out arow of picture elements (e.g., pixels), a subset of more than one rowsof picture elements, a single picture element, or some other non-rowgrouping of pixel elements, such as one or more columns of pictureelements. In some cases, such as where multiple pixel elements arereadout in parallel, for example, the readout time for a particulargroup (e.g., row or subset of rows of pixel elements) corresponds to thesame amount of time as that of a single picture element.

Generating Tracks of Closely-Timed Images Having Multiple ExposureLevels

FIG. 5 is a flow diagram illustrating an example process 500 forcreating a blended output track from multiple tracks of closely-timedexposure frames. As will be described, according to the process 500 andother techniques described herein, the blending module 117 generallyreceives and processes tracks of closely-timed exposure frames. Theblending module 117 may receive and or process the tracks on aframe-by-frame basis, or may operate on any portion of the tracks (e.g.,5, 10, 100, 1000 or more frames, or entire tracks) at any given time.

At block 502, the blending module 117 receives multiple tracks (orportions thereof) of video frames having closely timed exposures (e.g.,spaced apart by no more than about 20, 16, 7, 4, or 2 microseconds). Theblending module 117 can receive the tracks in a variety of ways. As oneexample, the blending module 117 resides in software executing on anexternal computing device. The camera 100 records the closely-timedexposure tracks to the memory device 114 and the tracks are transferredfrom the memory device 114 to the external computing device, whichforwards the recorded tracks to the blending module 117 for processing.As another example, the blending module 117 resides in software orhardware on the camera 100 and receives the recorded tracks directlyfrom the memory device 114 or sensor 112.

At block 504, the blending module 117 processes the received exposuresaccording to the desired (e.g., fixed or user-selectable) algorithm. Theblending module 117 may blend together the first and second tracksaccording to the algorithm, generating the output track based on thefirst and second (or more) tracks. Some example blending techniques areshown and described with respect to FIGS. 6A-9.

At block 506, the blending module 117 provides the blended output track.For example, where the blending module 117 resides on-camera, it mayprovide the output image stream frame to the memory 114 for storage.Alternatively, the camera may stream the blended image track to thecomputing device 102 for external processing and/or storage. Where theblending module 117 operates on only a frame or other portion of eachtrack at a time, the process 500 may then return to block 502, where theblending module 117 receives the first and second exposures for the nextframe or group of frames in the respective track.

FIGS. 6A-6B illustrate example blending operations. For example, theblending module 117 may be capable of implementing the blendingtechniques shown in FIGS. 6A-6B. Generally, using these and otherblending techniques, the blending module 117 is capable of providing anoutput track that incorporates desired content from the respectiveimages while discarding undesired image content. The discarded imagecontent can include noisy portions of one or more of the tracks, as wellas other portions that are not needed to provide the desired aestheticeffect. Moreover, in some cases the user can select which content toinclude in the output track, providing improved creative flexibility.

Referring to FIG. 6A, the blending operation 600 represents a scenarioin which the camera captures first (e.g., long) 604 and second (e.g.,short) 606 closely timed exposures, respectively, for each video frame.The width of the bars representing the long and short exposures 604, 606generally corresponds to the dynamic range of the sensor, which in turncan correspond to the range of possible pixel values for the sensor 112(e.g., 0 to 1023 for a 10-bit sensor, 0 to 4095 for a 12-bit sensor, 0to 65,535 for a 16-bit sensor, etc.). FIG. 6A corresponds to aconfiguration in which the output track has an expanded bit-depth ascompared to the sensor 112, and has an improved dynamic range ascompared to either of the individual tracks 604, 606 due to theselective inclusion of content from the two tracks 604, 606.

As shown, for each of the long and short exposures 604, 606 a certainnumber of captured values will be relatively noisy, having a relativelylower signal-to-noise ratio (SNR). For example, the SNR is relativelylow for lower pixel values and gradually increases as the measured pixelvalues increase. Moreover, the subject scene may include highlights incertain brighter regions. As indicated, the short exposure 606 cancapture a significant portion of the highlights because of the shorterintegration time, reducing the chance that the highlights will becomewashed out. On the other hand, as illustrated, the long exposure 604 insome cases does not capture much highlight detail due to the relativelylonger integration time, which can lead to wash out.

The scene may also include darker regions including shadows, which canoften be juxtaposed with brighter regions. As shown, the long exposure604 may capture relatively more detail in these regions due to the longintegration time. Conversely, the short exposure 606 may capture lessdetail in these regions due to the shorter integration time.

Also, as shown, the long and short exposures 604, 606 can also includecontent that corresponds to “normal” portions of the image scene. Theseportions may not be particularly dark or bright, for example, and canhave a generally normal or medium brightness level. The labels used withrespect to FIGS. 6A-6B (“shadows,” “normal,” “highlights”) are used onlyfor the purposes of illustrating the blending concepts, and are notlimiting. For example, it will be appreciated that the shorter exposuremay in some cases capture some amount of shadow content from the imagescene. Likewise, the longer exposure in some instances will record somehighlight detail.

In the illustrated example, the blending module 117 matches the lightlevels between the two exposures 604, 606. For example, the blendingmodule 117 can shift the exposures with respect to one another by N bits(e.g., 1, 2, 4, 8, or more bits) prior to blending, or perform someother mathematical operation on one or more of the exposures. In theillustrated example, the short exposure 606 is shifted up by N bits, andis therefore multiplied by a factor of 2^(N). In such cases, the longexposure 604 may be more exposed than the short exposure 606 by a factorof 2^(N). As such, after the shifting, the light levels between the twoexposures 604, 606 for corresponding image scene regions more closelymatch. Also, as shown, the noisiest content in the short exposure 606 isdiscarded in some cases.

A variety of other processes are possible for adjusting the exposures604, 606 with respect to one another prior to blending. For example, insome cases, the long exposure 604 is shifted or otherwise adjustedinstead of the shorter exposure 606. In one instance, the long exposure604 is shifted down by N bits instead of shifting the shorter exposure606 up by N bits. FIG. 3B, discussed below, shows one such embodiment.In other embodiments, the exposures 604, 606 are both shifted. Forexample, the exposures 604, 606 may be shifted in equal and oppositedirections. In yet other cases, the system shifts the exposures 604, 606in opposite directions but by different values. For example, where thefirst exposure 604 is exposed by a factor of 2^(N) greater than thesecond exposure 606, and where N=X+Y, the first exposure 604 may beshifted down by X bits, and the second exposure may be shifted up by Ybits. A variety of other embodiments are possible. For instance, insteadof multiplying or dividing the pixel values for the exposures 604, 606,a subtraction, addition or other appropriate operation can be used.

After matching the light levels, the blending module blends together theexposures 604, 606. Three blending algorithms (i)-(iv) are shown for thepurposes of illustration.

Referring to the first algorithm (i), the blending module 117 in oneembodiment selects the image content of the long exposure 604 thatcorresponds to the line segment A-B for inclusion in the output track.In one example, the sensor is a 12-bit sensor, the short exposure isshifted up by 2 bits, and the line segment A-B corresponds to pixelvalues between 0 and 2048. The blending module 117 uses the measuredvalues from long exposure as a reference to determine which pixel valuesto include in the blended output track. For example, the blending module117 determines which measured pixels in the long exposure 604 havevalues from 0 to 2048, and incorporates those pixel values into thecorresponding pixels in the blended output image. The values from thelong exposure 604 are incorporated substantially directly, unmodified,into the output track while, in some alternative configurations, theblending module 117 performs some mathematical operation on the valuesprior to incorporation, such as a scaling, shifting or rounding.

The charts 608, 610 shown for the long and short exposures 604, 606,respectively, indicate the percentage of each track included in theoutput track according to the blending algorithm (i). For example, forpixel values between points A and B, 100% of the output track iscomposed of content from the long exposure 604.

The blending module 117 may alternatively use the short exposure 606 asa reference to determine which measured pixel values from the longexposure 604 to include in the output track for the portion A-B. In theexample case, for instance, the blending module 117 determines whichpixels in the short exposure 606 have measured values between 0 and2048, and selects the measured values for the corresponding pixels inthe long exposure 604 to include in the output track. In yet otherinstances, the blending module 117 uses other types of references. Forexample, the blending module 117 may calculate some combination (e.g.,an average) of the measured values for the short and long exposure 606,604 for each pixel, and use the calculated value as a reference todetermine which pixels from the long exposure 604 to include in theoutput track. Similarly, while the remaining blending operations (e.g.,those along the lines B-C, C-D, and those for the other example blendingschemes (ii)-(iv)) are described as using the long exposure 604 as areference, the short exposure 606, some combination of the short andlong exposure 606, 604, or some other type of reference may be employedin those cases as well.

This image content from the long exposure 604 along the line segment A-Bmay include relatively good detail in the shadows. In some other cases,the blending module 117 blends together the short and long exposures604, 606 to some degree for values along the line A-B. The amount ofblending may be relatively constant, regardless of the input pixelvalues. Additionally, the blending may be biased in favor of one of thetracks, e.g., the long track 604. For example, the blending module 117may perform a weighted average of input values from the long track 604and the short track 606, where the long track has a higher weightingthan the short track 606. In some cases, the degree of bias or weightingis controllable by a user, or automatically adjusts depending on theparticular mode of operation. Blending algorithm (iv) described belowshows an example where weighted averages are used to generate some ofthe image content in the blended output track.

Referring still to algorithm (i), the next portion of the output trackcorresponds to a combination of the image content from both the long andshort exposures 604, 606 along the slanted line B-C. Moreover, thedegree to which the respective exposures 604, 606 are included in theoutput exposure varies based on the measured input values. This can bein contrast to the portions of the output image corresponding to theline segments A-B and C-D (discussed below). For example, for thesegments A-B and C-D, pixels from only one track are selected forinclusion in the output track, or the amount of blending is generallyconstant (e.g., weighted average), and irrespective of the measuredinput values.

A variety of operations are possible for the blending of the exposures604, 606 for the segment B-C. For example, in some embodiments, theamount of the long track 604 included in the output track is greatest atthe point B. Conversely, the amount of the short track 606 used in theblending operation is least at the point B. Moreover, for increasingpixel values x between B and C, the amount of the long track 604included in the output track gradually decreases. Conversely, the amountof the second track 606 included in the output track increases, reachinga maximum at the point C. This is illustrated in the chart 608, whichshows the percentage of the long exposure 604 in the output trackdecreasing linearly from 100 percent to 0 percent for increasing pixelvalues B to C. Conversely, the chart 610 shows the percentage of theshort exposure 606 in the output track increasing linearly from 0percent to 100 percent from B to C. In an example case, the sensor is a12-bit sensor, the short exposure 606 is shifted up by 2 bits, Bcorresponds to a pixel value of 2048, and C corresponds to a pixel valueof 4094. Taking as an example a point midway between B and Ccorresponding to a pixel value of 3071, the blending module 117determines which pixels in the long exposure 604 (the referenceexposure) have a value of 3071. Because the point is midway between Band C, the blending module 117 calculates the values for correspondingpixels in the output track according to the following equation:out_pixel_val=0.5*long_pixel_val*0.5short_pixel_val.  (Eq. 8)

A linear interpolation is used for the portion B-C in some instances,and in one case the interpolant is given by:

$\begin{matrix}{{y = {1 + \frac{x - B}{B - C}}},} & \left( {{Eq}.\mspace{14mu} 9} \right)\end{matrix}$where x corresponds to measured input pixel values between B and C, suchthat B≦x≦C, for example. In turn, the interpolant may be used tocalculate the interpolated output pixel value for each value of x. Forinstance, the output pixel values may be calculated according to thefollowing equation:out_pixel_val=y*long_pixel_val+(1−y)*short_pixel_val  (Eq. 10)

Other types of blending can be used for pixel values corresponding tothe line segment B-C. In some cases, and unlike the linear interpolationtechnique, the amount of the respective exposures 604, 606 included inthe blended exposure may not depend on the input values. For example,for this portion of the output track, the blending module 117 maycalculate an average of the image content from the long exposure 604 andthe image content of the short exposure 606, and include the average inthe output track. A variety of other operations can be used instead ofan average, including, without limitation, a sum, difference, weightedaverage, or the like. Blending algorithm (iii) described belowincorporates a curved blending function.

In one embodiment, the blending module 117 selects the image content ofthe short exposure 606 corresponding to the line C-D to complete theoutput track, and does not blend the exposures for this portion of theoutput track. For example, referring to 12-bit sensor example, the pointC may correspond to a pixel value of 4095, which corresponds to asaturated value for the long exposure 604. The blending module 117 maytherefore determine which pixels in the long exposure 604 have asaturated value of 4095 and incorporate into the output image the valuesfrom the corresponding pixels in the short exposure 606. As indicated,the image content along the line C-D includes relatively good highlightdetail. In other cases, some degree of blending may be used for theportion of the output track corresponding to the segment C-D. Forexample, for the portion C-D, the blending module 117 may perform aweighted average or other operation that is similar to those describedabove with respect to the segment A-B. However, the weighted average maybe biased in favor of the short track 606 instead of the long track 604for the portion C-D, unlike for the portion A-B. Algorithm (iv)described below incorporates a blending function along these lines.

Thus, according to such blending techniques, the blending module 117uses the short exposure 606 to preserve the highlights in the resultingoutput track, correcting for any possible loss of highlight detail. Thetrack that is made up of the short exposure frames may therefore bereferred to as a “highlight protection track” or “highlight correctiontrack,” for example.

As indicated by the arrows, one or more of the points A, B, C and D canbe adjusted to adjust the blending profile. This can greatly affect boththe dynamic range and the motion characteristic of the resulting outputtrack. For example, the lengths of the segments A-B and C-D cansignificantly alter the dynamic range of the image. Moreover, the motioneffect can be tuned by adjusting length of the segment B-C or theparticular blending operation (e.g., linear interpolation, weightedaverage, etc.) that is used on this portion. Specific motion controloperations are described in greater detail below with respect to FIG. 8.

Blending algorithm (ii) is similar to blending algorithm (i), exceptthat, as indicated by the shorter line segment B-C, the blending module117 blends together the long and short exposures 604, 606 for a muchsmaller portion of the output track. Moreover, as indicated by therelatively long line segment A-B, a larger portion of the long track 604is included directly in the output track as compared as compared toalgorithm (i).

As shown by the charts 608, 610 for algorithm (iii), a blending module117 implementing algorithm (iii) applies a curve on the measured inputvalues to generate the output track. While a variety of different curvesare possible, the blending module 117 in the illustrated example appliesan “S” curve over the portion of the output track corresponding to thepixel values between the points B and C. The curve may be a sigmoidcurve, or some other type of curve. As with the previous examples, oneor more of the points A, B, C, and D can be adjusted to change length ofthe various segments and corresponding blending profile. In oneinstance, the curve is applied over the entire output track.

There may additionally be more than three portions (A-B, B-C, C-D) ofthe output track having discrete blending profiles. Algorithm (iv) showssuch a case where there are separate blending profiles for fivedifferent portions of the output track: (1) A-B—direct incorporation ofthe long track 604; (2) B-C—weighted average, where the long track 604is weighted heavier than short track 606; (3) C-D—linear interpolationbetween long track 604 and short track 606; (4) D-E—weighted average,wherein the short track 606 is weighted heavier than the long track 604;and (5) E-F—direct incorporation of the short track 606.

Referring now to FIG. 6B, the illustrated blending operation is similarto that of FIG. 6A in that system again captures and blends togetherfirst (e.g., long) and second (e.g., long) closely timed exposures 604,606. However, unlike the operation shown in FIG. 6A, FIG. 6B shows aconfiguration where the blending module 117 does not expand thebit-width of the output track as compared to the input tracks. Rather,the bit-width remains the same. As one example, the sensor 112 is a12-bit sensor and thus outputs 12-bit long and short exposures 604, 606,although other bit-widths are possible. The blending module 117 performslight level matching by shifting the long exposure 604 down by a number(e.g., 2, 4, 8, etc.) of bits with respect to the long exposure 604.However, unlike the operation shown in FIG. 4A, the shifted portion ispreserved in the embodiment of FIG. 6B. The first and second tracks 604,606 are then blended together according to the selected algorithm,creating a 16-bit output track. Three example blending algorithms (i),(ii) and (iii) are shown, although a wide variety of other possibilitiesexist. As shown, the noisiest content in the long exposure 604 isdiscarded in some cases, while still preserving some detail in theshadows.

While described with respect to a number of specific examples in FIGS.6A-6B, the blending can occur in a variety of additional manners. Forexample, the blending can be based on contrast levels, as describedbelow with respect to FIG. 9.

FIG. 7 shows a blending process 700 for creating an output exposure frommultiple closely-timed exposures. Although described with respect tosingle first and second exposures for the purposes of illustration, theoperations described below with respect to the process 700 may insteadbe performed on entire tracks of first and second exposures or portionsthereof.

At blocks 702A, 702B, the sensor 112 captures first (e.g., long) andsecond (e.g., long) exposures. At block 704A, 704B, the blending module117 or other component processes the captured exposures.

For example, referring to FIGS. 6A-6B, in some cases one or more of theexposures 604, 606 is adjusted before blending, e.g., to match the lightlevels between the two exposures. In some of these cases, the blendingmodule 117 shifts one or both exposures 604, 606 and thereforemultiplies or divides the pixel values by a corresponding factor of two.However, sensor pixels in some cases output non-zero values even when nolight is incident on the pixels (e.g., when the lens cap is on thesensor), creating “non-zero black” values. It can desirable to take thisinto account prior to the blending. For example, multiplying these“non-zero black” values in the light matching process can complicatelater processing stages or produce other undesirable results.

Thus, in one embodiment the module 117 subtracts an offset (e.g., ablack offset) from the first and/or second exposures prior to the lightmatching. The module 117 then performs the light matching or othermathematical operation(s) on the exposures, e.g., multiplying each ofthe exposures by a predetermined factor. Finally, the blending module117 can add an offset to the first and second exposures, such as a blackoffset having the same magnitude as the value that was subtracted priorto the light matching.

In some configurations, at block 704A, 704B the blending module performsone or more operations on the first and second exposures to improve thecompression of the image data. For example, in one instance, the imageprocessing module 116 combines select (e.g., low) frequency componentsof wavelets from both the first and second exposures to improvecompression efficiency. The image processing module 116 may alsoimplement a pre-emphasis curve one or more of the exposures (e.g., thedarker, shorter exposure). Example pre-emphasis functions are describedin greater detail in the '967 patent (e.g., with respect to FIGS. 8, 8A,11 and columns 11-12 of the '967 patent) and are incorporated byreference herein. Such techniques can improve the efficiency or otherquality of the compression. For example, in some cases, pre-emphasis orother techniques can avoid or reduce the amount of compression thatoccurs on lower bits of the particular exposure(s).

At block 706, the blending module 117 receives parameters for use in theblending operation. For example, the blending parameters generallydefine how the first and second exposures will be combined. In somecases, for example, the system 100 provides an interface to set one ormore of the blending parameters. For example, the system 100 displays agraphical user interface (GUI) to the user that the user can interactwith to set the parameter(s). The GUI or other interface can be providedvia software on a display of an external computer, or may instead beprovided on the camera, depending on the configuration.

In one instance, the GUI includes a slider or one or more other iconsthat the user can manipulate to adjust the blending parameters. Thegraphic 707 is similar to those shown with respect to the algorithms ofFIGS. 6A-6B. Accordingly, the positions of “A,” “B,” “C,” and “D”generally define how to blend the first and second exposures (“EXP. 1,”“EXP. 2”). As indicated in FIG. 7, the user in some cases be able toadjust the position of “B” and “C” using the interface, thereby settinghow much of the first and second (or more) exposures to include in theresulting output track, and what portions of the tracks should beblended together. In some cases the user can adjust the position of “A”and/or “D” instead of or in addition to “B” and “C.” In addition to theillustrated example, a number of other compatible interfaces arepossible. Moreover, the interface can allow users to adjust a variety ofother parameters.

In another configuration, the user can select a “highlight protection”value, which determines the amount of highlight protection to employ.Referring to FIGS. 6A-6B, the selected “highlight protection” value maygenerally the exposure (e.g., integration time) of the exposures 606 inthe short track. In one embodiment, the user can select a number ofstops (e.g., 2, 3, 4, 5, or 6 stops) of highlight protection to include.For example, if the user selects 2 stops of highlight protection, theexposure for the short exposure 606 track will be 2 stops less than thatof the long track 604. For example, where the long track 604 exposurevalue is 1/48 seconds, the exposure for the short track 606 will be setto 1/192 seconds.

In another configuration, the system provides the user with a menu ofpre-set algorithms which each have a particular creative effect. Each ofthe pre-set algorithms have pre-determined blending parameters (e.g.,“B” and “C” values). The particular pre-set algorithms may alsodetermine what type of processing occurs in blocks 704A, 704B (e.g.,linear interpolation, weighted average, “S” curve, etc.) in some cases.

At block 708, the blending module 117 performs the blending operationbased on the selected blending parameters. For example, the blendingmodule 117 can perform the blending generally in the manner describedabove with respect to FIGS. 5-6B. Finally, at block 710, the blendingmodule 117 outputs the blended exposure (or track of exposures). Forexample, the blended exposures are provided for storage and/or playbackto the user.

In some embodiments, the system 100 provides the user with generallyreal-time feedback. For example, as the user changes one or moreblending parameters or selects a different pre-set blending algorithm,the system plays back the blended output track or a portion thereof foruser review on a camera or computer monitor. Thus, the user can makeadjustments on the fly to achieve the desired visual effect.

Returning to block 708, in some embodiments the blending module 117blends the closely-timed tracks by determining, according to theselected algorithm, which portions of each track are properly exposed.For example, for each exposure, the blending module 117 flags theproperly exposed pixels for inclusion in the combined output image.According to another embodiment, the blending module 117 compares pixelsfrom one exposure to corresponding pixel(s) in the other exposure(s),and according to a predetermined algorithm, calculates the correspondingpixel value in the blended output.

As described, any of the blending techniques can generally beimplemented on a frame-by-frame basis, for example, or at some othergranularity, and a wide variety of compatible blending techniques arepossible. In certain configurations, more than one frame from the firstand/or second (or more) tracks are compared or otherwise analyzed ingenerating the output track. For example, in such cases the blendingmodule 117 compares frames n−w, . . . , n, . . . , n+x from the firsttrack to frames n−y, . . . , n, . . . n+z from the second track togenerate frame n in the output image stream, where n is the currentoutput track frame.

Controlling Motion Effect

As discussed, the image capture techniques described herein efficientlyuse the frame period to provide tracks having different exposures levelswithin individual frames. The inter-exposure delay is relatively small,resulting in substantially reduced or no temporal gap between thedifferently exposed tracks. For example, there may be substantially novisual separation between objects in one track as compared to the sameobjects in another track. Moreover, as discussed, the inter-exposureskew for the individual tracks is relatively minimal. According tocertain aspects, the system exploits these and other aspects to controlthe motion effect in the combined output track, such as the amount orquality of blur, in addition to providing the dynamic range benefitsdescribed herein. For example, motion artifacts in the combined outputstream can be substantially reduced, removed, or controlled in a desiredfashion. Additionally, some techniques described herein reduce oreliminate the amount of motion artifact detection and processing thatcan be necessary with some conventional approaches.

FIG. 8 shows a process 800 of providing an output image track havingcontrolled motion effect. At block 802, the blending module 117 receivesat least two closely-timed tracks, which can be recorded according toany of the techniques described herein. For example, the tracks may beany of those shown and described in FIGS. 3A-6B. At block 804, theblending module 117 receives a set of motion control parameters. Theseparameters generally define the motion control algorithm that theblending module 117 performs when selectively combining tracks. As withthe dynamic range blending parameters (e.g., amount of “highlightprotection”) described above, the motion control parameters may beuser-selectable or fixed, depending on the implementation. In somecases, the user is provided a menu having a list of various motioncontrol techniques to choose from. These may be any of the techniquesdescribed below with respect to block 806, or some other technique(s).

At block 806, the process 800 combines the tracks of closely-timedimages to create an output image stream having the desired motioneffect. A variety of motion control techniques are possible.

Where multiple, tracks of closely-timed images are recorded at differentexposure levels, one of the tracks may be relatively blurry, similar toconventional video recordings. For example, significant blurring may bepresent in relatively long integration time tracks, such as those shownin FIGS. 3A-4C and 6A-6B. However, other tracks may be sharper andinclude reduced or minimal blurring. For example, the short integrationtime exposures shown in FIGS. 3A-4C and 6A-6B may be relatively sharper.Typical video recordings (e.g., 24 fps with a 180 degree shutter) show agenerally constant blurring for moving objects, without sharperreferences. However, what the human eye sees when viewing the scene inperson is closer to a combination of both the blurred references to theobjects as well as discrete, sharper references to the moving objects.

To mimic this effect, in some embodiments, the blending module 117 usesboth the more blurry track(s) and the sharper track(s) to control themotion effect, create an output track that includes both the blurred andsharp references to moving objects. For instance, referring again toFIGS. 6A-6B, the amount or quality of the motion effect included in theoutput track can be controlled by adjusting the blending parameters,such as the positions of A, B, C, D or the type of blending operationsused. As one example, a user can control the amount of blurred and sharpreferences represented in the output track by adjusting the position ofB and C, thereby adjusting the length and position of the line segmentsA-B, B-C, and C-D. In particular, referring to algorithm (i) shown inFIG. 6A, for increasing measured pixel values along the line B-C wherethe blending module 117 implements an interpolation function, the outputtrack will tend to include a varying mixture of blurred references andsharper references. Specifically, the output track along the line B-Cincludes an increasingly significant portion of image content from theshort track 606 and a correspondingly lesser portion of the long track604. As such, portions of the output track corresponding to the linesegment B-C will include a varying mixture of relatively more blurredreferences to image scene objects from the long track 604 and relativelysharper references to the same image scene objects from the short track606. In this manner, the blending module 117 can create an output trackthat more closely mimics what the human eye would see if watching thescene in person. The system or a user can additionally adjust thepositions of B and C to modify the effect.

In addition, gradually shifting the relative contributions of the largertrack 604 and the shorter track 606 (or vice versa) in an incrementalmanner, such as by using linear interpolation or a curve function, canhelp limit or prevent any banding or other undesirable visual artifactsthat may occur from a more abrupt change in the relative contributionsof the two tracks. In some cases, the gradual shifting can also help toreduce the amount of lower SNR image content that is incorporated in theoutput track (e.g., from the shorter track 606), improving imagequality.

In another example, the blending module 117 implements a weightedaverage of the long and short exposures 604, 606 for portions of theoutput track along the line B-C. For example, a 50/50 weighted averagemay result in an output track having a generally even mixture of sharpercontent from the short track 606 and blurred content from the longertrack 604. The weightings can be adjusted to tailor the effect. In oneembodiment, the system provides the user with an interface to controlhow much of the motion effect from each track to include in theresulting output track. Generally, the system can employ a wide varietyof other functions instead of or in combination with the illustratedones.

In yet further configurations, the blending module 117 can set therelative weighting of the short and long tracks 604, 606 in the outputtrack based on a contrast level. For instance, the blending module 117processes measured input values of the long track 604, the short track606, or a combination thereof to determine a contrast level inparticular image regions, and uses the determined contrast level toselect the relative weighting between the long track 604 and the shorttrack 606 for pixels within those image regions. For example, theblending module 117 applies a high-pass filter on the image data for theshort track 606 (and/or long track 606) to determine high contrastregions. For relatively high contrast regions, the blending module 117weights the short track 606 heavier than the long track 604 in theoutput track. For relatively lower contrast regions, the blending module117 may weight the long track 604 higher, or weight the two tracksequally. In this manner, the resulting output track can include bothsharp references to moving objects (from the high contrast regions ofthe short track 606) and blurred references to moving objects (fromadjacent, relatively lower contrast regions of the long track 604). Theblending module 117 can identify lower contrast regions instead of, orin addition to, higher contrast regions, by performing a low pass filteron the long and/or short exposures 604, 606, for example.

Moreover, for portions of the output track corresponding to the linesegments A-B and/or C-D, a user in some embodiments adjusts the motioneffect by controlling the amount of each image to include in the outputtrack. For example, the user can adjust the bias value in a weightedaverage blend operation to control the weight each input track is givenin the blending operation. For example, the system may provide a slider,other GUI, or another input mechanism to allow the user tailor theeffect.

In another embodiment, the blending module 117 is configured to blendthe tracks together in a manner that more closely matches the motioneffect of a traditional camera (e.g., 24 fps with a 180 degree shutter).For example, a customized motion estimation algorithm may be used tomatch the motion blur of one or more of the tracks to that of atraditional camera. For example, referring to FIG. 3A, the blur in theshorter integration time track 302 may be increased to match that of thelonger integration time track 304. In one embodiment, the blur matchingis done before the blending of the exposures, although it can be doneafterwards in other implementations.

At block 908, the blending module 117 outputs the motion-adjusted trackfor playback or further processing.

Terminology/Additional Embodiments

Embodiments have been described in connection with the accompanyingdrawings. However, it should be understood that the figures are notdrawn to scale. Distances, angles, etc. are merely illustrative and donot necessarily bear an exact relationship to actual dimensions andlayout of the devices illustrated. In addition, the foregoingembodiments have been described at a level of detail to allow one ofordinary skill in the art to make and use the devices, systems, etc.described herein. A wide variety of variation is possible. Components,elements, and/or steps can be altered, added, removed, or rearranged.While certain embodiments have been explicitly described, otherembodiments will become apparent to those of ordinary skill in the artbased on this disclosure.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orstates. Thus, such conditional language is not generally intended toimply that features, elements and/or states are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or states are included or are to beperformed in any particular embodiment.

Depending on the embodiment, certain acts, events, or functions of anyof the methods described herein can be performed in a differentsequence, can be added, merged, or left out all together (e.g., not alldescribed acts or events are necessary for the practice of the method).Moreover, in certain embodiments, acts or events can be performedconcurrently, e.g., through multi-threaded processing, interruptprocessing, or multiple processors or processor cores, rather thansequentially. In some embodiments, the algorithms disclosed herein canbe implemented as routines stored in a memory device. Additionally, aprocessor can be configured to execute the routines. In someembodiments, custom circuitry may be used.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. The described functionalitycan be implemented in varying ways for each particular application, butsuch implementation decisions should not be interpreted as causing adeparture from the scope of the disclosure.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein can be implementedor performed with a general purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general purpose processor can be a microprocessor, but in thealternative, the processor can be any conventional processor,controller, microcontroller, or state machine. A processor can also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

The blocks of the methods and algorithms described in connection withthe embodiments disclosed herein can be embodied directly in hardware,in a software module executed by a processor, or in a combination of thetwo. A software module can reside in RAM memory, flash memory, ROMmemory, EPROM memory, EEPROM memory, registers, a hard disk, a removabledisk, a CD-ROM, or any other form of computer-readable storage mediumknown in the art. An exemplary storage medium is coupled to a processorsuch that the processor can read information from, and write informationto, the storage medium. In the alternative, the storage medium can beintegral to the processor. The processor and the storage medium canreside in an ASIC. The ASIC can reside in a user terminal. In thealternative, the processor and the storage medium can reside as discretecomponents in a user terminal.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it will beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the spirit of the disclosure. As will berecognized, certain embodiments of the inventions described herein canbe embodied within a form that does not provide all of the features andbenefits set forth herein, as some features can be used or practicedseparately from others. The scope of certain inventions disclosed hereinis indicated by the appended claims rather than by the foregoingdescription. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

1. A method of obtaining video image data using a digital imaging sensorcomprising a picture element array of rows and columns, the methodcomprising: using a plurality of picture elements in the picture elementarray to capture light corresponding to a first image at a firstexposure level; using the plurality of picture elements in the pictureelement array to capture light corresponding to a second image at asecond exposure level different than the first exposure level;converting the light captured by the plurality of picture elements forthe first image to digital measurements; and converting the lightcaptured by the plurality of picture elements for the second image todigital measurements, wherein individual picture elements completecapturing the light corresponding to the first image prior to when thoseindividual picture elements begin to capture the light corresponding tothe second image, further wherein the step of using the plurality ofpicture elements in the picture element array to capture lightcorresponding to the second image at the second exposure level beginsprior to the completion of the step of converting the light captured bythe plurality of picture elements for the first image to digitalmeasurements, further wherein the step of converting the light capturedby the plurality of picture elements for the first image to digitalmeasurements substantially completes prior to the beginning of the stepof converting light captured by the plurality of picture elements forthe second image to digital measurements, further wherein the amount oftime that elapses from between when the step of capturing the lightcorresponding to the first image completes to when the step of capturingthe light corresponding to the second image completes is greater thanthe amount of time from between completing readout of a first of thepicture elements for the first image to completing readout ofsubstantially all of the picture elements for the first image, andfurther wherein first and second images are captured for a plurality ofvideo frames to create a first track of first images and to create asecond track of second images, where combined images are created for theplurality of video frames.
 2. The method of claim 1, wherein theplurality of picture elements form subsets of one or more rows ofpixels, and wherein the pixels in the respective subsets are capturedand converted as a group.
 3. The method of claim 1, wherein, for aparticular picture element of the plurality of picture elements, thereis an inter-exposure delay of less than approximately 100 pictureelement reset times between converting light captured by the particularpicture element for the first image and beginning to capture lightcorresponding to the second image at the second exposure level by theparticular picture element.
 4. The method of claim 3, wherein theinter-exposure delay is less than approximately 10 picture element resettimes.
 5. The method of claim 4, wherein the inter-exposure delay isapproximately one picture element reset time.
 6. The method of claim 1,wherein the first and second images are captured using a rolling shuttertechnique, wherein the capturing of light by a particular subset of oneor more rows of picture elements begins after the capturing of light bythe preceding subset of one or more rows of picture elements begins. 7.The method of claim 6, wherein an inter-picture element delay from whenthe capturing of light by a particular subset of one or more rows ofpicture elements begins and when the capturing of light by thesubsequent subset of one or more rows of picture elements begins is lessthan approximately two picture element read out times.
 8. The method ofclaim 7, wherein the inter-picture element delay is approximately onepicture element readout time.
 9. A method of blending video image datacorresponding to multiple exposures and captured using a digital imagingsensor comprising a picture element array of rows and columns,comprising: obtaining first image data corresponding to a first imageand derived from digital measurements that were converted from lightcaptured by a plurality of picture elements in the picture element arrayat a first exposure level; obtaining second image data corresponding toa second image and derived from digital measurements that were convertedfrom light captured by the plurality of picture elements in the pictureelement array at a second exposure level different than the firstexposure level; and using one or more processors to selectively combinethe first image data and the second image data to create combined imagedata corresponding to a combined image, wherein individual pictureelements completed capturing the light corresponding to the first imageprior to when those individual picture elements began to capture thelight corresponding to the second image, further wherein capturing ofthe light corresponding to the second image at the second exposure levelbegan prior to the completion of converting the light captured by theplurality of picture elements for the first image to digitalmeasurements, further wherein converting the light captured by theplurality of picture elements for the first image to digitalmeasurements substantially completed prior to the beginning ofconverting the light captured by the plurality of picture elements forthe second image to digital measurements, further wherein the amount oftime that elapsed from between when capturing the light corresponding tothe first image completed to when capturing the light corresponding tothe second image completed is greater than the amount of time frombetween completing readout of a first of the picture elements for thefirst image to completing readout of substantially all of the pictureelements for the first image, and further wherein first and secondimages are captured for a plurality of video frames to create a firsttrack of first images and to create a second track of second images,where combined images are created for the plurality of video frames. 10.The method of claim 9, wherein an exposure time for the second image islonger than an exposure time for the first image and the combined imagehas a wider dynamic range than either of the first and second images.11. The method of claim 9, wherein the first image has a firstbit-depth, the second image has a second bit-depth, and the combinedimage has a bit-depth that is greater than either of the first or secondbit-depths.
 12. The method of claim 9, further comprising blendingtogether content from the first image with content from the second imagefor at least some of the combined image, wherein portions of thecaptured image scene represented by the blended content include firstreferences from the first image to one or more image scene objects andsecond references from the second image to the same image scene objects,the second references depicting significantly more motion blur than thefirst references.
 13. The method of claim 9, further comprising blendingtogether content from the first image with content from the second imagefor at least some of the combined image by varying the relative amountsof first image content and second image content included in the blendedcontent in response to magnitudes of picture element values from one ormore of the first image and the second image.
 14. The method of claim13, wherein said varying comprises performing a linear interpolationfunction.
 15. The method of claim 13, wherein said varying comprisesperforming a weighted averaging function.
 16. The method of claim 13,further comprising adjusting the amount of blended content to include inthe combined image in response to user input.
 17. The method of claim13, wherein said blending together content comprises discardingrelatively low signal-to-noise content from one or more of the first andsecond images.
 18. An imaging system, comprising: a plurality of pictureelements arranged in an array; and control circuitry configured to:capture light corresponding to a first image at a first exposure levelwith the plurality of picture elements in the picture element array;capture light corresponding to a second image at a second exposure leveldifferent than the first exposure level with the plurality of pictureelements in the picture element array; convert the light captured by theplurality of picture elements for the first image to digitalmeasurements; and convert light captured by the plurality of pictureelements for the second image to digital measurements, whereinindividual picture elements complete capturing the light correspondingto the first image prior to when those individual picture elements beginto capture the light corresponding to the second image, further whereinthe control circuitry begins the capturing of the light corresponding tothe second image at the second exposure level prior to the completion ofthe converting of the light captured for the first image to digitalmeasurements, further wherein the control circuitry substantiallycompletes the converting of the light for the first image to digitalmeasurements prior to the beginning of the converting of the light forthe second image to digital measurements, further wherein the amount oftime that elapses from between when capturing the light corresponding tothe first image completes to when capturing the light corresponding tothe second image completes is greater than the amount of time frombetween completing readout of a first of the picture elements for thefirst image to completing readout of substantially all of the pictureelements for the first image, further wherein first and second imagesare captured for a plurality of video frames to create a first track offirst images and to create a second track of second images, wherecombined images are created for the plurality of video frames.
 19. Asystem for blending video image data corresponding to multiple exposuresand obtained using a digital imaging sensor comprising a picture elementarray of rows and columns, the system comprising: one or moreprocessors; an interface configured to: obtain first image datacorresponding to a first image and derived from digital measurementsthat were converted from light captured using a plurality of pictureelements in a picture element array of a digital imaging sensor at afirst exposure level; obtain second image data corresponding to a secondimage and derived from digital measurements that were converted fromlight captured using the plurality of picture elements in the pictureelement array at a second exposure level different than the firstexposure level; and a blending module executing on the one or moreprocessors and configured to selectively combine the first image dataand the second image data to create combined image data corresponding toa combined image, wherein individual picture elements completedcapturing the light corresponding to the first image prior to when thoseindividual picture elements began to capture the light corresponding tothe second image, further wherein the capturing of the lightcorresponding to the second image at the second exposure level beganprior to the completion of converting the light captured by theplurality of picture elements for the first image to digitalmeasurements, further wherein the converting the light captured by theplurality of picture elements for the first image to digitalmeasurements substantially completed prior to the beginning of the ofconverting the light captured by the plurality of picture elements forthe second image to digital measurements, further wherein the amount oftime that elapsed from between when capturing the light corresponding tothe first image completed to when capturing the light corresponding tothe second image completed is greater than the amount of time frombetween completing readout of a first of the picture elements for thefirst image to completing readout of substantially all of the pictureelements for the first image, and further wherein first and secondimages are captured for a plurality of video frames to create a firsttrack of first images and to create a second track of second images,where combined images are created for the plurality of video frames.